…stic#195492)

## Summary

Removed duplicated code cloud_security_posture_api_integration tests
folder


### Checklist

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

… gets cut off (elastic#195147)

## Summary

This PR fixes an issue where the Unified Field List field popover can
get cut off if its contents exceed the view height. Now, instead of
cutting off the popover, we limit the content height to `90vh` and make
the main section scrollable.

Before (from elastic#194313 test failure):

![image](https://github.com/user-attachments/assets/5927a899-a18a-4431-bd1d-6bb2682cd004)

After:

![scroll](https://github.com/user-attachments/assets/5071a52b-fbf4-4d05-96de-61858d9e5598)

Flaky test runs:
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7098

Fixes elastic#194313.
Fixes elastic#193934.
Fixes elastic#193781.

### Checklist

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <[email protected]>

…195461)

Closes elastic#194199

## Summary

Now that no plugins use anything from the `PresentationUtil` services
toolkit, it is safe to remove all code and documentation related to this
from the `PresentationUtil` plugin.


### Checklist

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

…rithms (elastic#193375)

## Summary

Completes elastic#190482


Switches rule `type` field to use the implemented diff algorithms
assigned to them in elastic#193369


Adds integration tests in accordance to
elastic#193372 for the `upgrade/_review`
API endpoint for the rule `type` field diff algorithm.

Also fixes some nested bracket misalignment that occurred in earlier PRs
with some test files

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

)

fix elastic#114562

![Screenshot 2024-10-08 at 09 04
20](https://github.com/user-attachments/assets/f43d592d-cc49-4f41-9bee-2a85f9c08292)

![Screenshot 2024-10-08 at 09 04
49](https://github.com/user-attachments/assets/39184960-d5ec-4485-944b-6ef4295d7101)

---------

Co-authored-by: Lisa Cawley <[email protected]>

## Summary

The enroll command must be executed after the elastic-agent is running.
This updates the instructions so that users are told to enable and start
the agent before running the `enroll` command to ensure the socket file
is created and available.

This fixes issues with errors like this:

```
{"log.level":"info","@timestamp":"2024-10-08T20:47:06.857Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://<REDACTED>.fleet.us-west-2.aws.elastic.cloud:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-08T20:47:08.681Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-08T20:47:08.683Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /usr/share/elastic-agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","ecs.version":"1.6.0"}
```

### Checklist

N/A

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

…hanges (elastic#190019)

## Summary
Issue: elastic#190018

Implement rule specific flapping support for create and update Rule API.
The new property on the rule is named `flapping`;

```
flapping: {
  look_back_window: number;
  status_change_threshold: number;
}
```

Also make changes in the task runner to use the rule's flapping settings
if it exists. Otherwise use the global flapping setting.

# To test
1. Go to
`x-pack/plugins/triggers_actions_ui/public/common/constants/index.ts`
and turn `IS_RULE_SPECIFIC_FLAPPING_ENABLED` to `true`
2. Create a rule with a rule specific flapping setting, generate the
alert and let it flap
3. Assert that the flapping is now using the rule specific flapping
4. Turn space flapping off
5. Assert that it no longer flaps despite having a rule specific
flapping
6. Try deleting/adding back the rule specific flapping via the UI and
verify everything works.

### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>

Generated by
https://buildkite.com/elastic/kibana-api-docs-daily/builds/855

## Summary

Addresses elastic#122070

## Summary

Follow up to elastic#194764. This test was also failing on 7.17 branch when
they 8.x ES compatibility tests were run, so this PR adjusts the test
based on the ES version it runs against. This will be backported to 8.x
and 7.17.

…lastic#195429)

Fixes elastic#191800

## Summary
Add missing privilege callout in Integrations Policies table. 
Currently the route
`app/integrations/detail/{pkgName}-{version}/policies` is available even
though the policies tab is not visible with limited privileges.

### Testing 
- Install `osquery_manager`
- Enable rbac feature flag
- Create role with privileges
![Screenshot 2024-10-08 at 16 24
46](https://github.com/user-attachments/assets/774de651-ac91-4365-9151-2df18efc811c)
- Log in with user with the above role
- Navigate to `app/integrations/detail/osquery_manager-1.14.0/policies`
- Verify that a limited privileges callout is displayed
![Screenshot 2024-10-08 at 16 12
23](https://github.com/user-attachments/assets/4498cbc1-243b-4fa9-a028-8899670f8e14)

elastic#194131

Use `entity.identityFields` instead of host, container and service
specific ones. Get the first environment available.

elastic#194519)

## 📓 Summary

Closes elastic#193319 
Closes elastic#193320 

This work is part of the effort to progressively deprecate the existing
Logs Stream feature.

Changes taken on this PR consist of:
- Create a new uiSettings `observability:enableLogsStream` which
defaults to `false` on the stateful/cloud deployments and is not
available in serverless ones (still, defaults to `false` behind the
scene).
- When `observability:enableLogsStream` is `false`, the Logs Stream page
route is not registered, and neither is its deep link for global search.
- When `observability:enableLogsStream` is `false`, the panels list on
Dashboard won't show anymore the option `Logs Stream (Deprecated)` to
prevent usage of this embeddable in new dashboards. The embeddable is
still registered for retro-compatibility with active dashboards, and it
has now a callout explaining the status of this embeddable
(unmaintained/deprecated).
- Rename logs ML to "Logs Anomalies" and "Logs Categories".

Other minor improvements regard:
- Remove duplicate Xstate utils and use the relative package instead.
- Remove the duplicated `useBoolean` hook used in the deprecation
callout.
- Sync deep links registration with available routes through a single
`getLogsRoutes` util.

## 🎥 Recordings

### Logs Stream app removed


https://github.com/user-attachments/assets/f4173294-8789-4abd-9972-29c9b7c197ed

### Logs Stream dashboard panel entry removed


https://github.com/user-attachments/assets/7f99ca2a-c030-4867-b976-8fdc1df09d29

### Logs Stream app removed from project nav


https://github.com/user-attachments/assets/de51bdd6-820a-4c03-8b64-fb1a6ced0a12

### Embeddable deprecation callout

<img width="949" alt="Screenshot 2024-10-02 at 10 22 09"
src="https://github.com/user-attachments/assets/99fd5554-004b-45e4-81db-cb23947e210e">

### Unavailable setting in serverless


https://github.com/user-attachments/assets/91bf6c37-0845-4918-a485-b6250bbd96bf

---------

Co-authored-by: Marco Antonio Ghiani <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Mike Birnstiehl <[email protected]>

## Summary

Fix for: [elastic#187962](elastic#187962)
We were displaying ES|QL based searches in various areas of ML, not just
in the places mentioned in the issue.
Before:
![Screenshot 2024-10-04 at 16 16
26](https://github.com/user-attachments/assets/dff7e1d6-4c8e-4916-acec-c6b9931c2a39)
Then, after selecting the ESQL based search:

![image](https://github.com/user-attachments/assets/9314cd0b-442a-4287-9d29-799e172f929a)
After the fix:

![image](https://github.com/user-attachments/assets/e660ef24-c585-4d95-bcf1-2578ec9e663d)

…ll screens due to lack of vertical scrolling (elastic#195234)

## Summary

Closes elastic#184048

- the rule type selection modal is now scrollable on smaller windows 


https://github.com/user-attachments/assets/47082b35-02a7-4b67-9a88-ee4200908bef

Co-authored-by: Antonio <[email protected]>

…c#194614)

## Summary

- Updates Trained Models table layout 
- Adds  the E5 model disclaimer 
- Removes redundant success toasts about model download, deletion, and
start of a deployment

<img width="1504" alt="image"
src="https://github.com/user-attachments/assets/e151afca-a9bf-4b4e-9d8c-a87c86c83ef9">

### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

…astic#194896)

## Summary

Related issues:
-  elastic#18511
-  elastic#192301

In this PR, in Job management > expanded row > Forecasts tab - a delete
action has been added to each row in the forecasts table. A confirmation
modal allows the user to confirm the delete action.

In the SMV view, the forecast being currently viewed is now highlighted
in the Forecast modal to make it easier to identify.


![image](https://github.com/user-attachments/assets/87814889-d41d-4780-98ab-695c6f12a023)

<img width="881" alt="image"
src="https://github.com/user-attachments/assets/accbd7d9-1bae-4f1f-af8f-8bd36eae0572">

<img width="1099" alt="image"
src="https://github.com/user-attachments/assets/6011936d-3773-41ce-bbce-3ca4c0154cab">

Dark mode:

<img width="882" alt="image"
src="https://github.com/user-attachments/assets/cbec6fc8-0c62-4e34-9546-0124ae80a568">


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>

…stic#194866)

## Summary

This an attempt to fix flaky Cypress test:
https://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main

I was not able to reproduce it locally.
Also, old version of test was very stable on flaky test runner too:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078

I changed test a bit by re-arranging order of form filling. So,
suppression fields will be last. Maybe it can reduce possibility of race
condition when form is just rendered and fields being interacted with by
Cypress.
Also, added assertion if threshold checkbox changed it status to enabled
before interacting with other suppression inputs.
If this won't help, next step can be using default suppression
configuration instead.

New version of test: 200 runs w/o failures
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084

---------

Co-authored-by: Ryland Herrick <[email protected]>

## Summary
Kibana-related pipelines are hard to find on Buildkite, due to other,
ingest-related pipelines having 'kibana' in their names.

This pipeline adds tags to pipelines serving `kibana` CI duties, so they
can be easily found using Buildkite's tags/labels.

The tags added are mostly `kibana` but some pipelines also got the
`security-solution` label, as these pipelines can be easily associated
with the served solution.

…95433)

## Summary

add locator to link to data stream management recently made available
elastic#195299

…favour of an internal one (elastic#194829)

New internal GET `/api/endpoint/metadata/transforms` route.

Current public GET `/api/endpoint/metadata/transforms` route is set to
deprecated.

All usages across the project have been updated to consume the new
internal route.

---------

Co-authored-by: kibanamachine <[email protected]>

Co-authored-by: elastic-renovate-prod[bot] <174716857+elastic-renovate-prod[bot]@users.noreply.github.com>

## Release Notes

Automatic Import is more forgiving if an LLM returns an ECS mapping in a
slightly unexpected format.

## Summary

When implementing elastic#194386 an issue
has been encountered where Claude returns the field name `date_format`
instead of expected `date_formats` and the ECS chain breaks down.

We add this case as a test to
`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.

Without the changes in this PR the list returned by
`findInvalidEcsFields` is

```
      [
        'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',
        'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',
        'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',
        'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',
        'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',
        'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',
        'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',
        'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'
      ]
```

while with these changes the result does not contain any `Invalid ECS field` messages.

The key changes are in the `processMapping` function:

1. We made function more forgiving in regards to the input, accepting
`date_format` in lieu of `date_formats`.
2. We have removed the collection of "other paths", that is, the reverse
index for simple values like `0.8`.

The latter change generally limits the impact of any other format issues
in the ECS mapping in the future.

Additionally, the function has been renamed to `extractECSMapping`, its
output type validated, and documentation has been added.

---------

Co-authored-by: Elastic Machine <[email protected]>

…ransformational commands (elastic#195863)

## Summary

Closes elastic#195752

This PR is fixing 2 bugs:

- It filters out counter fields from the breakdown as they are not
supported. I created a new util for this
- Fixes a bug unrelated with the breakdown (it also exists in previous
minors). The LensVis service is computing suggestions and pushes them to
`availableSuggestionsWithType `. In some indexes (it depends on the
types of the first 5 columns of the index) the lens suggestions api
might return a suggestion. So in that case the array has the histogram
suggestion + the suggestion from the suggestions api. So the service
will pick the first one which is not the histogram. But we know that in
case of non transformational commands we want to suggest the histogram.
So this PR is fixing it by ensuring that the array is cleaned up before
pushing the histogram suggestion.


Note: The 2 bugs are unrelated I just decided to fix them in one PR as
they are both histogram bugs.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

…#195556)

## Summary

This PR is a follow-up of elastic/pull/193966 and adds: 

1. Datastreams filter to data usage metrics page.
2. Metrics filter (hidden for now) that lists out metric types to
request.
3. Refactors to make code easier to maintain.
4. Shows a callout if no data stream is selected.

### screen
![Screenshot 2024-10-09 at 17 36
32](https://github.com/user-attachments/assets/a0779c91-25ae-4a64-819e-bc8a626f1f96)

### clip

![latest-metrics-ux](https://github.com/user-attachments/assets/0f4b1a9b-d160-435b-917b-f59c3a5cc9f8)

### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

---------

Co-authored-by: kibanamachine <[email protected]>

<img width="1616" alt="Screenshot 2024-10-10 at 10 04 51"
src="https://github.com/user-attachments/assets/b543a156-ea5e-46ba-9460-e86d7ca6e5a1">
<img width="1600" alt="Screenshot 2024-10-10 at 10 05 24"
src="https://github.com/user-attachments/assets/a2d7973f-53b3-4bf9-a917-8ce496d3c943">

---------

Co-authored-by: jennypavlova <[email protected]>
Co-authored-by: kibanamachine <[email protected]>

…#195990)

## Summary

Sets the correct capabilities for Onboarding cards:

- Integrations: 
- capability: `fleet.read`: The only privilege a user needs to access
the Integrations page, it won't be able to install anything though.
(`fleet` is the id for "Integrations" capability, the one for "Fleet" is
`fleetv2`).

- Dashboards: 
  - capability: `dashboard.show`

- AI Assisant: 
  - capability: `securitySolutionAssistant.ai-assistant`, 
  - license: `enterprise`

- Attack Discovery (still hidden): 
  - capability: `securitySolutionAttackDiscovery.attack-discovery`, 
  - license: `enterprise`

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Angela Chuang <[email protected]>
Co-authored-by: Agustina Nahir Ruidiaz <[email protected]>

## Summary

Removes SCSS files for the Single Metric Viewer and adds BEM classes for
`annotations`.
Affects the Single Metric Viewer in ML and the embeddable.
Part of [elastic#140695](elastic#140695)

…lastic#195405)

## 📓 Summary

Browsing fields from the Discover sidebar, I noticed integration fields
never show a related description even if they exist. The same is
happening in the fields table for the document detail flyout.

This happens due to `integration` and `dataset` parameters not being
passed to the service.


https://github.com/user-attachments/assets/0946cc71-44fb-4fc7-8e9d-b146bdd811f2

These changes improve the resolution of the integration field metadata:

- The `integration` and `dataset` params are no longer required to
attempt resolving and integration field metadata.
They are still accepted as an explicit hint in case we cannot infer
correctly some integration packages from the field name.
- The above change enables querying fields from different integrations
and datasets at once, enabling metadata retrieval for mixed data
sources.
- The integration retrieved from the EPR is now cached with its relevant
version, solving a potential corner case as explained
[here](elastic#183806 (review)).


https://github.com/user-attachments/assets/ae9cafd8-2581-4ce0-9242-cbb4e37c7702

---------

Co-authored-by: Marco Antonio Ghiani <[email protected]>

## Release Notes

Automatic Import can now create integrations for logs in the CSV format.
Owing to the maturity of log format support, we thus remove the verbiage
about requiring the JSON/NDJSON format.

## Summary

**Added: the CSV feature**

The issue is elastic#194342 

When the user adds a log sample whose format is recognized as CSV by the
LLM, we now parse the samples and insert the
[csv](https://www.elastic.co/guide/en/elasticsearch/reference/current/csv-processor.html)
processor into the generated pipeline.

If the header is present, we use it for the field names and add a
[drop](https://www.elastic.co/guide/en/elasticsearch/reference/current/drop-processor.html)
processor that removes a header from the document stream by comparing
the values to the header values.

If the header is missing, we ask the LLM to generate a list of column
names, providing some context like package and data stream title.

Should the header or LLM suggestion provide unsuitable for a specific
column, we use `column1`, `column2` and so on as a fallback. To avoid
duplicate column names, we can add postfixes like `_2` as necessary.

If the format appears to be CSV, but the `csv` processor returns fails,
we bubble up an error using the recently introduced
`ErrorThatHandlesItsOwnResponse` class. We also provide the first
example of passing the additional attributes of an error (in this case,
the original CSV error) back to the client. The error message is
composed on the client side.

**Removed: supported formats message**
 
The message that asks the user to upload the logs in `JSON/NDJSON
format` is removed in this PR:

<img width="741" alt="image"
src="https://github.com/user-attachments/assets/34d571c3-b12c-44a1-98e3-d7549160be12">


**Refactoring**
 
The refactoring makes the "→JSON" conversion process more uniform across
different chains and centralizes processor definitions in
`.../server/util/processors.ts`.

Log format chain now expects the LLM to follow the `SamplesFormat` when
providing the information rather than an ad-hoc format.
 
When testing, the `fail` method is [not supported in
`jest`](https://stackoverflow.com/a/54244479/23968144), so it is
removed.

See the PR for examples and follow-up.

---------

Co-authored-by: Elastic Machine <[email protected]>

…h policy (remove magic number) (elastic#195961)

elastic#195904)

Adds a new `MlAuditLogger` service for logging calls to elasticsearch in
kibana's audit log.
Not all calls are logged, only ones which make changes to ML jobs or
trained models, e.g. creating, deleting, starting, stopping etc.

Calls to the es client are wrapped in a logging function so successes
and failures can be caught and logged.

the audit log can be enabed by adding this to the kibana yml or dev.yml
file
`xpack.security.audit.enabled: true`

An example log entry (NDJSON formatted to make it readable):
```
{
  "event": {
    "action": "ml_start_ad_datafeed",
    "type": [
      "change"
    ],
    "category": [
      "database"
    ],
    "outcome": "success"
  },
  "labels": {
    "application": "elastic/ml"
  },
  "user": {
    "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
    "name": "elastic",
    "roles": [
      "superuser"
    ]
  },
  "kibana": {
    "space_id": "default",
    "session_id": "U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68="
  },
  "trace": {
    "id": "4f1b616b-8535-43e1-8516-32ea9fe76d19"
  },
  "client": {
    "ip": "127.0.0.1"
  },
  "http": {
    "request": {
      "headers": {
        "x-forwarded-for": "127.0.0.1"
      }
    }
  },
  "service": {
    "node": {
      "roles": [
        "background_tasks",
        "ui"
      ]
    }
  },
  "ecs": {
    "version": "8.11.0"
  },
  "@timestamp": "2024-10-11T09:07:47.933+01:00",
  "message": "Starting anomaly detection datafeed datafeed-11aaaa",
  "log": {
    "level": "INFO",
    "logger": "plugins.security.audit.ecs"
  },
  "process": {
    "pid": 58305,
    "uptime": 100.982390291
  },
  "transaction": {
    "id": "77c14aadc6901324"
  }
}
```

---------

Co-authored-by: kibanamachine <[email protected]>

…c#193968)

## Summary

Removing the indices stats tiles as requested and agreed as a part of
Ent Search deprecation here:
elastic/search-team#8231

![CleanShot 2024-09-25 at 12 54
16@2x](https://github.com/user-attachments/assets/bd8ee089-2bee-4beb-927b-e937975d8dbc)

---------

Co-authored-by: Elastic Machine <[email protected]>

…lastic#196057)

## Summary

I realized that as part of this
[PR](https://github.com/elastic/kibana/pull/192805/files#diff-8f26b8327cc9fc31bef2b22bb53b82256edc9cf05cfc9c766d746a7aa4532437L144),
`getIsActive` method was accidentally removed from `Applications` and
`Infrastructure` menus. This PR brings `getIsActive` back. I didn't find
any bug with the absence of `getIsActive`. Purpose of this PR is to not
remove something that was there before.

…ile first loading (elastic#195777)

## Summary

Makes the loading state and empty state mutually exclusive in the
grouping component to avoid showing the empty state when first loading
the groups data.

## To verify

1. Create one or more O11y rules that fire alerts
2. Open the O11y > Alerts page
3. Toggle on grouping
4. Reload the page (possibly after activating network throttling)
5. Verify that while the loading indicator is shown, the empty state is
not and viceversa

## References

Fixes elastic#190954

## Summary
The generated version of the docker image builder script didn't have
timeouts between retries, so a temporary outage on `docker.elastic.co`
would cause a docker pull error, failing the build (see:
https://buildkite.com/elastic/kibana-artifacts-snapshot/builds/4845#01927b40-43f9-471e-9e2c-407320fac978)

This PR adds a fix 15s per retry to the docker build runner.

…serverless (elastic#195763)

Fixes elastic#195599

## Summary

This PR ensures that we can use templates in the case action when:
1. the project is serverless security, and
2. the rule is created in stack management

### How to test

1. Add the following line to `serverless.yml` -
`xpack.cloud.serverless.project_id: test-123`
3. Start Elastic search in serverless security mode - `yarn es
serverless --projectType security`
4. Start Kibana in serverless security mode - `yarn start
--serverless=security`
5. Go to Security > Cases > Settings and add a template.
6. Go to stack and create a rule with the case action.
7. Confirm the template created in step 5 can be selected.

<img width="586" alt="Screenshot 2024-10-10 at 15 00 46"
src="https://github.com/user-attachments/assets/5379e1d1-f0eb-4829-9604-ee5a0e3d050b">

**Please double-check also that the templates in the case action still
work as expected in normal scenarios.**

---------

Co-authored-by: kibanamachine <[email protected]>

…sconfiguraiton flyout (elastic#195932)

## Summary

Fixes:
- elastic/security-team#10793

`v96.1.0`⏩`v97.0.0`

_[Questions? Please see our Kibana upgrade
FAQ.](https://github.com/elastic/eui/blob/main/wiki/eui-team-processes/upgrading-kibana.md#faq-for-kibana-teams)_

---

## [`v97.0.0`](https://github.com/elastic/eui/releases/v97.0.0)

**Breaking changes**

- EuiDataGrid's custom grid body (rendered via `renderCustomGridBody`)
no longer automatically renders the column header row or footer rows. It
instead now passes the `headerRow` and `footerRow` React elements, which
require manual rendering.
([elastic#8028](elastic/eui#8028))
- This change was made to allow consumers to sync header/footer rows
with their own custom virtualization libraries.
- To facilitate this, a `gridWidth` prop is now also passed to custom
grid body renderers.

**Bug fixes**

- Fixed inputs not taking the whole width when passing `fullWidth` as
`true` to EuiDatePickerRange component
([elastic#8061](elastic/eui#8061))

**Accessibility**

- Improved accessibility of `EuiExternalLinkIcon` by clarifying text for
Screen Reader users. ([elastic#8065](elastic/eui#8065))

---------

Co-authored-by: Elastic Machine <[email protected]>

…instead become disabled (elastic#194743)

## Summary

This PR sets the Web Crawler tile to point out the external Open Web
Crawler repo when there is no ent-search node running rather than become
disabled using the `crawlerDisabled`

Before:

![CleanShot 2024-10-02 at 18 25
57@2x](https://github.com/user-attachments/assets/2cffe7c8-fbb1-4192-956f-69ba8ec5529a)

After:

![CleanShot 2024-10-02 at 18 25
11@2x](https://github.com/user-attachments/assets/fcf7ac0f-2985-4b7a-9100-3968054505c7)


Also the empty state of Web crawler points out to the Source code repo
when there is no ent-search instance running using the
`errorConnectingMessage`. This improvement should fix this issue
elastic/search-team#8319

![CleanShot 2024-10-08 at 11 48
44@2x](https://github.com/user-attachments/assets/1dedc24e-e23a-4188-a676-f910a9b2ce6c)


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Elastic Machine <[email protected]>

elastic#196077)

## Summary

The index details page is always updated even when the plugin is
disabled. Using the pluginEnabled conditional to only update when
enabled.

### How to replicate
1. disable uisetting for search indices plugin
2. go to index management and click on a index detail

Expected: see the old index detail page
actual: goes to the new index detail url but does not render the search
detail page (as plugin disabled)

### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

…dges (elastic#195351)

Closes elastic/observability-dev#4007

Updates integration badges according to [the latest
designs](https://www.figma.com/design/CPhMyRNOgo0wsEiaIMZJ14/Onboarding-Quick-Starts?node-id=3015-58062&t=5tvnrPIOkfg7xAJp-1).

![CleanShot 2024-10-08 at 08 57
48@2x](https://github.com/user-attachments/assets/1ae0d9d7-3b1f-4acd-b16f-ad1cbc09db88)

## Summary

`POST /api/alerting/rule/{id}/_mute_all` in
elastic#195181

Closes [elastic#192115](elastic#192115)
Closes [elastic#192465](elastic#192465)


## Summary

This PR adds synthrace client for Otel native data and a simple
scenario. This is the first step of adding it and in the future it will
include more metrics and use cases.

>[!NOTE]
> To run ES the command needs "xpack.otel_data.registry.enabled=true"
flag
> `yarn es snapshot --license trial --E
"xpack.otel_data.registry.enabled=true"`

## Next steps
- We currently have only `service_destination` in the metrics indices we
can include the other types in the future
- After we have all the UI changes we can add more scenarios (also using
the opentelemetry demo data and not only the e2e PoC example)

## Testing
- Run ES: 
```bash 
yarn es snapshot --license trial --E "xpack.otel_data.registry.enabled=true"
```
- Run Kibana:
```bash 
yarn start
```

>[!WARNING]
If the e2e PoC is used the first 2 steps should be skipped

- Run syntrace: 
```bash
node scripts/synthtrace otel_simple_trace.ts --clean
```
- Check indices in DevTools for the generated data: 
```bash 
GET *metrics-generic.otel*/_search

GET *traces-generic.otel*/_search

GET *logs-generic.otel*/_search
```
- Check in the APM UI (all the tabs) 
>[!WARNING]
Currently the UI changes done in APM are not merged so some errors are
expected)


https://github.com/user-attachments/assets/92f63610-82da-40f3-89bb-00be83c55377

---------

Co-authored-by: miriam.aparicio <[email protected]>

…astic#196025)

…ests for `cases` and `alerting` custom RBAC

…grations-poc

…igration